A Comparison of Two Smoothing Methods for Word Bigram Models

Word bigram models estimated from text corpora require smoothing methods to estimate the probabilities of unseen bigrams. The deleted estimation method uses the formula: Pr(i j j) = fi + (1 )fijj ; where fi and fijj are the relative frequency of i and the conditional relative frequency of i given j, respectively, and is an optimized parameter. MacKay (1994) proposes a Bayesian approach using Dirichlet priors, which yields a di erent formula: Pr(i j j) = Fj + mi + 1 Fj + fijj where Fj is the count of j and and mi are optimized parameters. This thesis describes an experiment in which the two methods were trained on a two-million-word corpus taken from the Canadian Hansard and compared on the basis of the experimental perplexity that they assigned to a shared test corpus. The methods proved to be about equally accurate, with MacKay's method using fewer resources. ii Acknowledgements I would like to thank my supervisors, Graeme Hirst and Dekai Wu. Dekai suggested the topic of this thesis and provided technical guidance, over and above his duties at the Hong Kong University of Science and Technology. Graeme coached me through departmental procedures, corrected the writing style, and exhibited patience and optimism. David MacKay helpfully explained the details of his method, and advised me regarding the implementation. He also served as the external reader. Bill Gale, Radford Neal, and Peter Brown answered statistical questions. Finally, I would like to thank my family, James and Michael, for their love and encouragement, and God, who makes all things possible.